Data augmentation has become an important part of modern deep learning pipelines and is typically needed to achieve state of the art performance for many learning tasks. It utilizes invariant transformations of the data, such as rotation, scale, and color shift, and the transformed images are added to the training set. However, these transformations are often chosen heuristically and a clear theoretical framework to explain the performance benefits of data augmentation is not available. In this paper, we develop such a framework to explain data augmentation as averaging over the orbits of the group that keeps the data distribution approximately invariant, and show that it leads to variance reduction. We study finite-sample and asymptotic empirical risk minimization and work out as examples the variance reduction in certain two-layer neural networks. We further propose a strategy to exploit the benefits of data augmentation for general learning tasks.

Data augmentation has become an important part of modern deep learning pipelines and is typically needed to achieve state of the art performance for many learning tasks. It utilizes invariant transformations of the data, such as rotation, scale, and color shift, and the transformed images are added to the training set. However, these transformations are often chosen heuristically and a clear theoretical framework to explain the performance benefits of data augmentation is not available. In this paper, we develop such a framework to explain data augmentation as averaging over the orbits of the group that keeps the data distribution approximately invariant, and show that it leads to variance reduction. We study finite-sample and asymptotic empirical risk minimization and work out as examples the variance reduction in certain two-layer neural networks. We further propose a strategy to exploit the benefits of data augmentation for general learning tasks.

The idea of learning representations in hyperbolic spaces has rapidly gained prominence in the last decade, attracting a lot of attention and motivating extensive investigations. This rise of interest was partly launched by statistical-physical studies [1] which have shown that distinctive properties of complex networks are naturally preserved in negatively curved continuous spaces. Since complex networks are ubiquitous in modern science and everyday life, this relation with hyperbolic geometry provided a valuable hint for low-dimensional representations of hierarchical data [2]. More generally, structural information of any hierarchical data set may be better represented in negatively curved manifolds rather than in flat ones. This further implies that hyperbolic geometry provides a suitable framework for simultaneous learning of hypernymies, similarities and analogies. This hypothesis triggered the interest of many data scientists and machine learning (ML) researchers in hyperbolic geometry. Nowadays, hyperbolic ML is a rapidly developing young subdiscipline within the broader field of geometric deep learning [3].